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Abstract 

We give necessary and sufficient conditions for solvability of Aj = XWjX' , with the 
Aj are m given positive semi-definite matrices of order n. The solution A is n x p and 
the m solutions Wj are required to be diagonal, positive semi-definite, and adding up 
to the identity. We do not require that p < n. 
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Note: This is a working paper which will be expanded/updated frequently. All suggestions 
for improvement are welcome. The directory deleeuwpdx.net/pubfolders/simul has a pdf 
version, the complete Rmd hie, and the bib hie. 


1 Introduction 

There is by now a large literature on simultaneous diagonalization of matrices. The topic is of 
importance in blind source separation, chemometrics, psychometrics, nonlinear optimization, 
multilinear algebra, fast matrix multiplication, and quantum physics. We have no intention of 
giving a complete overview of the literature, our purpose is mainly to present and generalize 
the approach of De Leeuw and Pruzansky (1978). 

The problem is the following. If A 1? • • ■ , A m are real symmetric matrices of order n , then 
we want to solve the system of equations Aj = XWjX', where X is any n x p matrix and 
where the Wj are diagonal of order p. In this paper we will restrict our attention to the case 
in which the Aj are positive semi-dehnite and in which we require the Wj to be positive 
semi-dehnite as well. There are several important special cases that can be distinguished. 
If p < n then Schonemann (1972) presented the main algebraic result, in the context of 
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psychometric data analysis. His computational approach, however, left something to be 
desired, and consequently it was improved, generalized, and implemented by De Leeuw and 
Pruzansky (1978). We also note the special case in which m = 2, in which some additional 
results can be obtained (De Leeuw (1982)). 

In this paper we generalize the approach of Schonemann (1972) and De Leeuw and Pruzansky 
(1978) to the case where we do not necessarily have p < n. As in the more restrictive case, we 
start by studying the decomposition Aj = XWjX' without the requirement that the Wj are 
diagonal. We give necessary and sufficient conditions for the set of solutions to be non-empty, 
and we describe the complete set of solutions in the case in which the system is solvable. We 
then characterize the solutions, if any, in the set of all solutions which have diagonal Wj. 


2 1-rank 


Start with a convenient dehntion of the rank of a sequence of positive semi-definite matrices. 

Defintion 1: [lrank] Suppose Aj are m positive semi-definite matrices of order n. The 
tuple (Ai, ■ ■ ■ , A m ) has l-rank less than or equal to p if there is an n x p matrix X and there 
are p x p positive semi-definite matrices Wj, with sum IT. = /, such that Aj = XWjX' . It 
has 1-rank equal to p if it has 1-rank less than or equal to p, but not less than or equal to 
p — 1. Note that we allow for p > n. 


It may not be immediately obvious why we require IT. = I. Suppose IT. has rank r < p. Its 
eigen decomposition is 
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with E diagonal and positive definite or order r. Write the Wj in the form 

r , In. R.1 T at 1 1 

Wj = 

Thus C,, the sum of the Cj, is zero. And because the Cj are positive semi-definite this means 
all Cj are zero. Also, lemma 1 tells us that if a diagonal element of a positive semi-definite 
matrix is zero, then the whole corresponding row and column are zero. Thus the Bj are 
zero as well. It follows that XWjX' = (XN)Dj(N'X') and thus (Ai,--- , A m ) has 1-rank 
less than or equal to r < p. Thus it makes sense to require IT. to be at least, positive 
definite. But then, using for example the Cholesky decomposition IT. = SS', we see that 

)(S'X'), so we may as well require IT. = I. 
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XWjX' = (XS)(S 

Theorem 1: [lrank] (A x 


A m ), with sum A ., has 1-rank r : = rank(A.) 


Proof: Suppose the 1-rank of (A\, • • • , A m ) is p < r. Then A, = XX ', for some n x p matrix 
X, and rank(A.) = rank(A") < p < r, a contradiction. Thus l-rank(A], • • • , A m ) > r. 


To show l-ra.nk(A 1 , • • • ,A m ) < r we will actually construct the set of all solutions to 
Aj = XWjX' with IT. = /. Start with the eigenvalue decomposition of A,. 
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where A 2 is diagonal and positive definite of order r. 

Because A, = XX' it follows from lemma 2 that X = KAL', with a p x r orthonormal L. It 
remains to solve Aj = KAL'WjLAK'. Write 


Now A,K± = 
Consequently 
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where the Uj and Vj are arbitrary, except for the fact that we must have Wj positive semi- 
definite, and we must have U. = 0 and V, = I. Remember that L and L± are also arbitrary, 
except for their ortho normality. One convenient solution for Wj, with minimal Frobenius 
norm, is Wj = LA~ 1 K'AjKA~ 1 = X + Aj(X + )' , where X + is the Moore-Penrose inverse of 

X. QED 


3 d-rank 

We now turn to the case in which the Wj are required to be diagonal, and they still have to 
add up to the identity. 

Defintion 2: [drank] The sequence (A\, ■ ■ ■ , A m ) has d-rank less than or equal to p if there 
is an n x p matrix X and there are p x p diagonal positive semi-definite matrices Wj, with 
sum W, = I, such that Aj = XWjX'. It has d-rank equal to p if it has d-rank less than or 
equal to p, but not less than or equal to p — 1. Note that again we allow for p > n. 

Clearly 1-rank is less than or equal to d-rank, which implies by theorem 1 that d-rank is 
larger than or equal to r = rank(A»). In order to proceed we need another definition. 

Defintion 3: [pextension] Any p x p symmetric matrix of order p which has the r x r 
matrix A as its leading principal submatrix is a p-extension of A. 

Theorem 2: [drank] The d-rank of (Ai, ■ ■ ■ .A m ) is less than or equal to p if and only if 
there exist for each j — !,■■■ , rn pairwise commuting positive semi-definite p-extensions Cj 
of Bj = A~ 1 K'AjKA~ 1 that add up to one. 

Proof: Equation (1) in the proof of theorem 1 shows there must exist Uj and Vj such that 


A~ 1 K'A j KA~ 1 
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But this is the same as saying there must be p-extensions of the Bj that commute. QED 
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It seems that in general for p > n and m > 2 commuting p-extensions are difficult to work 
with. But in some cases theorem 2 simplifies. 

Corollary 1: [small_d_rank] The d-rank of (A±, ■ ■ ■ , A m ) is r = rank(A») if and only if 

AjA'Ac = A^A'Aj for all j,l. 

Proof: The A~ 1 K'AjKA~ 1 must commute without any p-extension. This translates to the 
condition in the theorem. QED 

Finally, we also have as a corollary a version of the basic result of De Leeuw (1982). 
Corollary 2: [m=2] The d-rank of (Ai,A 2 ) is rank^x + A 2 ). 

Proof: Because A~ l K'AiKA~ l + A~ 1 K , A 2 KA~ 1 = I we see from lemma 4 that 
A~ X K'A X K A” 1 and A~ 1 K'A 2 KA~ 1 commute. QED 


4 Appendix: Some Lemmas 


Lemma 1: [diagonal] If A is positive semi-definite and an = 0 then a t j = aji = 0 for all j. 


Proof: Suppose 
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is positive semi-definite. Define z = yi —er 
If r'Sr = 0 and r'r > 0 then z'Az < 0 for al 
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with e > 0. Then z'Az = —2 er'r + e 2 r'Sr. 
e > 0, which contradicts that A is positive 
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which again contradicts that A is positive semi-definite. Thus r'r = 0, i.e. r = 0. QED 


Lemma 2: [crossprod] Suppose the positive semi-definite matrix A of order n has eigenvalue 
decomposition 
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with A 2 a positive definite diagonal matrix of order r = rank(A). The equation A = XX', 
with X an n x p matrix has a solution if and only if p > r. All solutions are of the form 
X = KAL', with L is a p x r orthonormal matrix. 


Proof: Write X as 


which gives 
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Thus XX' = A, if and only if V = 0 and UU' = A. It follows that X = KAL' , with a p x r 
orthonormal L. Also rank(A") = rank(A») = r and p > r. QED 
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Lemma 3: [simultaneous] Suppose (Ai, • • • , A m ) is a sequence of real symmetric matrices 
of order n. Then there exist a square orthonormal X and diagonal W 3 such that Aj = XWjX' 
if and only if the Aj commute in pairs, i.e. if and only if AjA i = AgAj for all j ^ t. 

Proof: It is obvious that simultaneously cliagonalizability implies that the Aj commute in 
pairs. 


The interesting part of the proof is to show that pairwise commuting implies simultaneous 
cliagonalizability. The standard proof, repeated most recently in Jiang and Li (2016), uses 
induction, starting from the fact that the lemma is trivially true for m = 1. We give the 
proof in our notation and make it perhaps a bit more explicit and computational. 

So let us suppose the real symmetric matrices (Ax, • • • , A m ) commute in pairs. And suppose 
A m has eigenvalue decomposition 
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with all A s different. Set K : — 
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A m AjK s — AjA m K s — X\AjK s 


and thus AjK s are eigenvectors of A m with eigenvalue A s , i.e. AjK s 
this as 


K'AjK 
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K s (K' s AjK s ). Write 


0 0 • • • K' r AjK r 


Now obviously the matrices K'AjK commute in pairs, which implies that that the m — 1 
matrices (K' S A\K S1 ■ ■ ■ , K' s A m _iK s ) commute in pairs for each s. By the induction hypothesis 
there are square orthonormal L s and diagonal such that K' s AjK s = L s <b ]S L' S . Define 
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QED 

Lemma 4: [commute] If A and B are symmetric matrices with A + B — I then A and B 
commute. 

Proof: AB = A(I — A) = A — A 2 , which is symmetric. Another way to see this is that A 
and B = I — A have the same eigenvectors L , and L diagonalizes both matrices. QED 
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